Feedback Guided Load Balancing in Distributed Memory Environments ——————————– Masters Dissertation

Any code that parallelizes computational work is required to come up with a method to load balance the work across the available processors. Most codes of this nature use some heuristic to decide how to perform this load balancing. In essence, they must have a (cheap) method of coming up with a number to act as a proxy for dividing up the actual computational work. For some applications that use these codes, that heuristic works very well; for others, the performance achieved is less optimal. This project implements a dynamic load balancing algorithm for use in application problems executed on a distributed memory architecture. The idea is relatively simple: during simulation, record load data during computation time on each processor when each round of the program loop executes. Use the local load data and total load data to approximate the actual workload across each processor. Take that approximation against the total workload and redistribute the load across processors accordingly to achieve a better balance for the next run. Repeat the prior steps through every round of the program loop. If the workload is reasonable, this method converges to an optimal distribution of work, leading to a well load-balanced program execution. This project implements and benchmarks such a dynamic load balancing algorithm in MPI to analyze the performance in a distributed memory scenario and the effect that such dynamic load balancing has. Benchmarking this algorithm involves using computational work for a variety of different workload distributions to see how the algorithm handles different imbalances in workload that occur during program initialization and/or execution. The computational work used in benchmarking can be obtained by either using real application data to simulate the computational work, or by writing test functions to fake real computational work. Given that obtaining real application data to fit all test cases is difficult, test methods are used to fill the scenarios to simulate. These benchmarked results are then analyzed to determine the overall performance of the new algorithm given the overall increased availability of processors to execute over. In general, the results of the performance tests have shown that the new algorithm converges to an optimal solution in a minimal number of load balancing steps, dependent upon the load characteristics of the problem. In addition, the time taken for each load balance step, and the time for the total execution of the problem are reasonably low and do not vary much, even when increasing the total problem size or scaling over more processors. Those results are important in showing the viability of the algorithm for use in real problem situations. The ability to scale over a larger number of processors without letting the increased communication negatively affect performance show the algorithm to be a possible replacement for existing shared and distributed memory dynamic load balancing techniques among certain classes of problems.